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A  Manager's  Checklist  for  Validating  Software  Cost 
and  Schedule  Estimates 


Abstract.  This  report  provides  a  checklist  of  questions  to  ask  and  evidence  to 
look  for  when  assessing  the  credibility  of  a  software  cost  and  schedule  estimate. 
The  checklist  can  be  used  either  to  review  individual  estimates  or  to  motivate  and 
guide  organizations  toward  improving  their  software  estimating  processes  and 
practices. 


1.  introduction 


When  you — or  your  boss  or  customer — receive  a  cost  or  schedule  estimate  for  a  software 
project,  what  do  you  look  for  to  determine  your  willingness  to  rely  on  that  estimate? 

This  report  provides  a  checklist  to  help  you  address  and  evaluate  a  number  of  issues  that  are 
key  to  estimates  we  can  trust.  The  checklist  guides  you  through  the  seven  questions  in  this 
table: 


Seven  Questions  to  Ask  When 
Assessing  Your  Willingness  to  Rely 
_ On  a  Cost  and  Schedule  Estimate 

1 .  Are  the  objectives  of  the  estimate  clear  and  correct? 

2.  Has  the  task  been  appropriately  sized? 

3.  Are  the  estimated  cost  and  schedule  consistent  with 
demonstrated  accomplishments  on  other  projects? 

4.  Have  the  factors  that  affect  the  estimate  been  identified  and 
explained? 

5.  Have  steps  been  taken  to  ensure  the  integrity  of  the 
estimating  process? 

6.  Is  the  organization's  historical  evidence  capable  of  supporting 
a  reliable  estimate? 

7.  Has  the  situation  changed  since  the  estimate  was  prepared? 


Each  question  is  illustrated  with  elements  of  evidence  that,  if  present,  support  the  credibility 
of  the  estimate.  The  answers  you  receive  should  help  you  judge  the  extent  to  which  you  can 
safely  use  the  estimate  for  planning,  tracking,  or  decision  making.  The  checklist  can  be  used 
also  to  motivate  and  guide  organizations  toward  improving  their  software  estimating 
processes  and  practices. 
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2.  The  Validation  Checklist 


We  present  the  checklist  for  validating  software  cost  and  schedule  estimates  on  the  pages 
that  follow.  You  may  make,  use,  or  distribute  as  many  copies  as  you  wish,  so  long  as  you 
reproduce  the  entire  checklist,  including  the  copyright  notice. 

The  purpose  of  the  checklist  is  not  to  impose  criteria,  but  to  arm  you  with  questions  to  ask 
when  deciding  whether  or  not  to  rely  on  a  particular  estimate.  Only  you  can  determine  which 
questions  to  ask  and  whether  or  not  the  answers  you  get  provide  sufficient  evidence  to 
satisfy  your  requirements  for  using  the  estimate. 

Although  we  prepared  this  checklist  to  help  you  evaluate  estimates  for  software  costs  and 
schedules,  almost  everything  in  it  applies  equally  to  hardware  and  systems  engineering 
projects.  If  you  have  responsibilities  for  developing  hardware  or  integrated  systems,  you  may 
find  that  altering  the  word  'software'  wherever  it  appears  will  make  the  checklist  applicable  to 
estimates  for  these  systems  as  well. 

The  checklist  in  this  report  was  produced  as  part  of  the  SEI's  Software  Cost  Estimating 
Improvement  Initiative  [Park  94].  Please  let  us  know  if  you  find  it  helpful,  or  if  you  have 
suggestions  for  improving  its  usefulness  for  your  organization.  For  related  checklists  that  can 
help  you  evaluate  the  processes  used  for  making  software  estimates,  please  see  the  SEI 
report  Checklists  and  Criteria  for  Evaluating  the  Cost  and  Schedule  Estimating  Capabilities  of 
Software  Organizations  [Park  95]. 
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A  Manager's  Checklist 

for  Validating  Software  Cost  and  Schedule  Estimates 

This  checklist  is  designed  to  help  managers  assess  the  credibility  of 
software  cost  and  schedule  estimates.  It  identifies  seven  issues  to 
address  and  questions  to  ask  when  determining  your  willingness  to 
accept  and  use  a  software  estimate.  Each  question  is  associated 
with  elements  of  evidence  that,  if  present,  support  the  credibility  of 
the  estimate. 


Issue  1.  Are  the  objectives  of  the  estimate  clear  and 
correct? 

Evidence  of  Credibility 

The  objectives  of  the  estimate  are  stated  in  writing. 

The  life  cycle  to  which  the  estimate  applies  is  clearly  defined. 

The  tasks  and  activities  included  in  (and  excluded  from)  the 
estimate  are  clearly  identified. 

The  tasks  and  activities  included  in  the  estimate  are  consistent 
with  the  objectives  of  the  estimate. 


Issue  2.  Has  the  task  been  appropriately  sized? 

Evidence  of  Credibility 

A  structured  process  has  been  used  to  estimate  and  describe 
the  size  of  the  software  product. 

A  structured  process  has  been  used  to  estimate  and  describe 
the  extent  of  reuse. 

The  processes  for  estimating  size  and  reuse  are  documented. 

The  descriptions  of  size  and  reuse  identify  what  is  included  in 
(and  excluded  from)  the  size  and  reuse  measures  used. 

The  measures  of  reuse  distinguish  between  code  that  will  be 
modified  and  code  that  will  be  integrated  as-is  into  the  system. 

The  definitions,  measures,  and  rules  used  to  describe  size  and 
reuse  are  consistent  with  the  requirements  (and  calibrations)  of 
the  models  used  to  estimate  cost  and  schedule. 
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The  size  estimate  was  checked  by  relating  it  to  measured  sizes  □ 
of  other  software  products  or  components. 

The  size  estimating  process  was  checked  by  testing  its  □ 

predictive  capabilities  against  measured  sizes  of  completed 
products. 


Issue  3.  Are  the  estimated  cost  and  schedule  consistent 
with  demonstrated  accomplishments  on  other 
projects? 

Evidence  of  Credibility 

The  organization  has  a  structured  process  for  relating  estimates 
to  actual  costs  and  schedules  of  completed  work. 

•  The  process  is  documented. 

•  The  process  was  followed. 

The  cost  and  schedule  models  that  were  used  have  been 
calibrated  to  relevant  historical  data. 

(Models  of  some  sort  are  needed  to  provide  consistent  rules 
for  extrapolating  from  previous  experience.) 

The  cost  and  schedule  models  quantify  demonstrated  organiza¬ 
tional  performance  in  ways  that  normalize  for  differences  among 
software  products  and  projects. 

(So  that  a  simple,  unnormalized,  iines-of-code  per  staff- 
month  extrapolation  is  not  the  basis  for  the  estimate.) 

The  consistency  achieved  when  fitting  the  cost  and  schedule 
models  to  historical  data  has  been  measured  and  reported. 

The  values  used  for  cost  and  schedule  model  parameters 
appear  valid  when  compared  to  values  that  fit  the  models  well  to 
past  projects. 

The  calibration  of  cost  and  schedule  models  was  done  with  the 
same  versions  of  the  models  that  were  used  to  prepare  the 
estimate. 

The  methods  used  to  account  for  reuse  recognize  that  reuse  is 
not  free. 

(The  estimate  accounts  for  activities  such  as  interface 
design,  modification,  integration,  testing,  and  documentation 
that  are  associated  with  effective  reuse.) 


□ 

□ 

□ 

□ 


□ 


□ 

□ 

□ 

□ 


Page  2  of  6 


Extrapolations  from  past  projects  account  for  differences  in 
application  technology. 

(For  example,  data  from  projects  that  implemented 
traditional  mainframe  applications  require  adjustments  if 
used  as  a  basis  for  estimating  client-server  implementation. 
Some  cost  models  provide  capabilities  for  this,  others  do 
not.) 

Extrapolations  from  past  projects  account  for  observed,  long¬ 
term  trends  in  software  technology  improvement. 

(Although  some  cost  models  attempt  this  internally,  the  best 
methods  are  usually  based  on  extrapolating  measured 
trends  in  calibrated  organizational  performance.) 

Extrapolations  from  past  projects  account  for  the  effects  of 
introducing  new  software  technology  or  processes. 

(Introducing  a  new  technology  or  process  can  initially 
reduce  an  organization's  productivity.) 

Work-flow  schematics  have  been  used  to  evaluate  how  this 
project  is  similar  to  (and  how  it  differs  from)  projects  used  to 
characterize  the  organization's  past  performance. 


Issue  4.  Have  the  factors  that  affect  the  estimate  been 
identified  and  explained? 

Evidence  of  Credibility 

A  written  summary  of  parameter  values  and  their  rationales 
accompanies  the  estimate. 

Assumptions  have  been  identified  and  explained. 

A  structured  process  such  as  a  template  or  format  has  been 
used  to  ensure  that  key  factors  have  not  been  overlooked. 

Uncertainties  in  parameter  values  have  been  identified  and 
quantified. 

A  risk  analysis  has  been  performed,  and  risks  that  affect  cost  or 
schedule  have  been  identified  and  documented. 

(Elements  addressed  include  issues  such  as  probability  of 
occurrence,  effects  on  parameter  values,  cost  impacts, 
schedule  impacts,  and  interactions  with  other 
organizations.) 


Issue  5.  Have  steps  been  taken  to  ensure  the  integrity  of 
the  estimating  process? 

Evidence  of  Credibility 

Management  reviewed  and  agreed  to  the  values  for  all 
descriptive  parameters  before  costs  were  estimated. 

Adjustments  to  parameter  values  to  meet  a  desired  cost  or 
schedule  have  been  documented. 

If  a  dictated  schedule  has  been  imposed,  the  estimate  is 
accompanied  by  an  estimate  of 

•  The  normal  schedule. 

•  The  additional  expenditures  required  to  meet  the  dictated 
schedule. 

Adjustments  to  parameter  values  to  meet  a  desired  cost  or 
schedule  are  accompanied  by  management  action  that  makes 
the  values  realistic. 

More  than  one  cost  model  or  estimating  approach  has  been 
used,  and  the  differences  in  results  have  been  analyzed  and 
explained. 

People  from  related  but  different  projects  or  disciplines  were 
involved  in  preparing  the  estimate. 

At  least  one  member  of  the  estimating  team  is  an  experienced 
estimator,  trained  in  the  cost  models  that  were  used. 

Estimators  independent  of  the  performing  organization  concur 
with  the  reasonableness  of  the  parameter  values  and  estimating 
methodology. 

The  groups  that  will  be  doing  the  work  accept  the  estimate  as  an 
achievable  target. 

Memorandums  of  agreement  have  been  completed  and  signed 
with  the  other  organizations  whose  contributions  affect  cost  or 
schedule. 


Issue  6.  Is  the  organization's  historical  evidence  capable 
of  supporting  a  reliable  estimate? 

Evidence  of  Credibility 

The  estimating  organization  has  a  method  for  organizing  and 
retaining  information  on  completed  projects  (a  historical 
database). 

The  database  contains  a  useful  set  of  completed  projects. 

Elements  included  in  (and  excluded  from)  the  effort,  cost, 
schedule,  size,  and  reuse  measures  in  the  database  are  clearly 
identified. 

(See,  for  example,  the  SEI  checklists  for  defining  effort, 
schedule,  and  size  measures.) 

Schedule  milestones  (start  and  finish  dates)  are  described  in 
terms  of  criteria  for  initiation  or  completion,  so  that  work 
accomplished  between  milestones  is  clearly  bounded. 

Records  for  completed  projects  indicate  whether  or  not  unpaid 
overtime  was  used. 

Unpaid  overtime,  if  used,  has  been  quantified,  so  that  recorded 
data  provide  a  valid  basis  for  estimating  future  effort. 

Cost  models  that  were  used  for  estimating  have  been  used  also 
to  provide  consistent  frameworks  for  recording  historical  data. 

(This  helps  ensure  that  comparable  terms  and  parameters 
are  used  across  all  projects,  and  that  recorded  data  are 
suitable  for  use  in  the  estimating  models.) 

The  data  in  the  historical  database  have  been  examined  to 
identify  inconsistencies,  and  anomalies  have  been  corrected  or 
explained. 

(This  is  best  done  with  the  same  cost  models  that  are  used 
for  estimating.) 

The  organization  has  a  structured  process  for  capturing  effort 
and  cost  data  from  ongoing  projects. 

The  producing  organization  holds  postmortems  at  the 
completion  of  its  projects. 

•  To  ensure  that  recorded  data  are  valid. 

•  To  ensure  that  events  that  affected  costs  or  schedules  get 
recorded  and  described  while  they  are  still  fresh  in  people's 
minds. 


Information  on  completed  projects  includes 

•  The  life-cycle  model  used,  together  with  the  portion  covered 
by  the  recorded  cost  and  schedule. 

•  Actual  (measured)  size,  cost,  and  schedule. 

•  The  actual  staffing  profile. 

•  An  estimate  at  completion,  together  with  the  values  for  cost 
model  parameters  that  map  the  estimate  to  the  actual  cost 
and  schedule. 

•  A  work  breakdown  structure  or  alternative  description  of  the 
tasks  included  in  the  recorded  cost. 

•  A  work-flow  schematic  that  illustrates  the  software  process 
used. 

•  Nonlabor  costs. 

•  Management  costs. 

•  A  summary  or  list  of  significant  deliverables  (software  and 
documentation)  produced  by  the  project. 

•  A  summary  of  any  unusual  issues  that  affected  cost  or 
schedule. 

Evolution  in  the  organization's  work-flow  schematics  shows 
steady  improvement  in  the  understanding  and  measurement  of 
its  software  processes. 


Issue  7.  Has  the  situation  changed  since  the  estimate  was 
prepared? 

Evidence  of  Credibility 

The  estimate  has  not  been  invalidated  by  recent  events, 
changing  requirements,  or  management  action  (or  inaction). 

The  estimate  is  being  used  as  the  basis  for  assigning  resources, 
deploying  schedules,  and  making  commitments. 

The  estimate  is  the  current  baseline  for  project  tracking  and 
oversight. 
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